Systems and methods for providing a quiescing protocol

ABSTRACT

The systems and methods of the present invention provide a quiescing protocol. In one embodiment, nodes of a system utilize the protocol to complete processing until they reach a consistent state. In one embodiment, a coordinator initiates the quiescing process and the nodes communicate with each other to determine whether their messages have been processed and communicate with the coordinator to determine when all of the messages have been processed.

REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims benefit of priorityunder 35 U.S.C. §120 to U.S. patent application Ser. No. 11/357,740,filed Feb. 17, 2006, titled “SYSTEMS AND METHODS FOR PROVIDING AQUIESCING PROTOCOL”, which is hereby incorporated by reference herein inits entirety.

LIMITED COPYRIGHT AUTHORIZATION

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF THE INVENTION

This invention relates to systems and methods for providing a quiescingprotocol that allows a set of nodes to continue processing until theyreach a consistent state.

BACKGROUND

The increase in processing power of computer systems has ushered in anew era in which information is accessed on a constant basis. Oneresponse has been to distribute processing requests across multiplenodes or devices. A distributed architecture allows for more flexibleconfigurations with respect to factors such as speed, bandwidthmanagement, and other performance and reliability parameters.

The distributed architecture also allows for a more fault-tolerantsystem. Accordingly, if one of the nodes is not available, then theother nodes may continue to process requests. Additional problems mayoccur, however, when one of the failed nodes comes back online and triesto reintegrate into the system.

Because of the foregoing challenges and limitations, there is an ongoingneed to improve the manner in which nodes of a distributed architectureprocess requests.

SUMMARY OF THE INVENTION

The systems and methods described herein provide a quiescing protocol.

In one embodiment, a distributed system configured to process messageswithin the distributed system is provided. The distributed system mayinclude a plurality of distributed nodes configured to permitcommunication among the plurality of nodes and to process messages; atleast one of the plurality of nodes acting as a coordinator, thecoordinator configured to initiate a quiescing protocol and to determinewhether relevant messages have been processed by the plurality of thedistributed nodes.

In an additional embodiment, a method of quiescing a set of messagesprocessed within the system is provided. The method may include sendingstart messages to a plurality of nodes indicating that a quiescingprotocol is to begin; receiving checkpoint messages from the pluralityof nodes indicating that the plurality of nodes have received theirprobe response messages; sending continue messages to the plurality ofnodes requesting received message information from the plurality ofnodes; receiving received message information from the plurality ofnodes; and determining that the received message information indicatesthat the plurality of nodes have not received any new messages.

In a further embodiment, a system for quiescing a set of messagesprocessed within the distributed system is provided. The system mayinclude means for sending start messages to a plurality of nodesindicating that a quiescing protocol is to begin; means for receivingcheckpoint messages from the plurality of nodes indicating that theplurality of nodes have received their probe response messages; meansfor sending continue messages to the plurality of nodes requestingreceived message information from the plurality of nodes; means forreceiving received message information from the plurality of nodes; andmeans for determining that the received message information indicatesthat the plurality of nodes have not received any new messages.

In an additional embodiment, a method of quiescing a distributed systemis provided. The method may include setting a counter and countingmessages that have been received; ensuring that all messages previouslysent have been processed by their destination; performing a checkpoint;and performing a second checkpoint using the value of the counter.

In a further embodiment, a method of quiescing a distributed system isprovided. The method may include receiving a start message from acoordinator indicating that a quiescing protocol is to begin; suspendingnew messages; setting a received message information to indicate thatthere are no messages received; receiving messages from the plurality ofnodes; updating the received message information to indicate that amessage has been received; sending probe messages to the plurality ofnodes; receiving probe response messages from the plurality of nodes;receiving probe messages from the plurality of nodes; for each probemessage received from the plurality of nodes, sending correspondingprobe response messages to the plurality of nodes; sending a checkpointmessage to the coordinator indicating that the probe response messageshave been received from the plurality of nodes; receiving a continuemessage from the coordinator requesting the received messageinformation; sending the received message information to thecoordinator; and receiving a done message from the coordinator.

For purposes of summarizing this invention, certain aspects, advantagesand novel features of the invention have been described herein. It is tobe understood that not necessarily all such advantages may be achievedin accordance with any particular embodiment of the invention. Thus, theinvention may be embodied or carried out in a manner that achieves oroptimizes one advantage or group of advantages as taught herein withoutnecessarily achieving other advantages as may be taught or suggestedherein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B illustrate embodiments of high-level block diagrams ofone embodiment of nodes in a system.

FIG. 2 illustrates an exemplary timing chart of communication amongnodes using one embodiment of a quiescing protocol.

FIG. 3 illustrates one embodiment of a high-level block diagram of oneembodiment of a node in the system.

FIG. 4 illustrates one embodiment of a flow chart of a quiescingprotocol for a coordinator.

FIGS. 5A, 5B, and 5C illustrate embodiments of flow charts of aquiescing protocol for a participant.

FIGS. 6A, 6B, and 6C illustrate embodiments of a high-level blockdiagram of one embodiment of set of smart storage units in a distributedfile system.

FIG. 7 illustrates one embodiment of a flow chart for mapping a changeto resource assignments in a distributed system.

These and other features will now be described with reference to thedrawings summarized above. The drawings and the associated descriptionsare provided to illustrate embodiments of the invention and not to limitthe scope of the invention. Throughout the drawings, reference numbersmay be re-used to indicate correspondence between referenced elements.In addition, the first digit of each reference number generallyindicates the figure in which the element first appears.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Systems and methods which represent one embodiment and exampleapplication of the invention will now be described with reference to thedrawings. Variations to the systems and methods which represent otherembodiments will also be described.

For purposes of illustration, some embodiments will be described in thecontext of a distributed file system. The present invention is notlimited by the type of environment in which the systems and methods areused, however, and the systems and methods may be used in otherenvironments, such as, for example, the Internet, the World Wide Web, aprivate network for a hospital, a broadcast network for a governmentagency, an internal network of a corporate enterprise, an intranet, alocal area network, a wide area network, and so forth. Some of thefigures and descriptions, however, relate to an embodiment of theinvention wherein the environment is that of a distributed file system.It is also recognized that in other embodiments, the systems and methodsmay be implemented as a single module and/or implemented in conjunctionwith a variety of other modules and the like. Moreover, the specificimplementations described herein are set forth in order to illustrate,and not to limit, the invention. The scope of the invention is definedby the appended claims.

I. OVERVIEW

Distributed architectures require the cooperation of many differentnodes. Nodes are constantly communicating with each other in order, forexample, to share common resources and to divide responsibilities. Nodessend messages to other nodes, and these messages may trigger thecreation of other messages, and so forth. There may be many exampleswhen it is desirable to stall the state of the distributed architecturein order to perform some system operation. Because messages sent toother nodes may trigger a long line of succeeding messages, it isdifficult to know when a system has reached a consistent state and,thus, has been successfully quiesced.

Accordingly, there is a need for a quiescing protocol for a distributednetwork system in order to perform maintenance to or to monitor thesystem. For example, it may be desirable to quiesce a distributed systemin order to take a global snapshot of data structures and/or othersystem information, which may be used for diagnosing or debuggingpurposes. Alternatively, it may be desirable to quiesce a distributedsystem in order to change the responsibilities of different nodes. Forexample, if one node is removed from or loses communication with thesystem, there is a need to reassign the responsibilities of the lostnode to the remaining nodes. In order to do this effectively andefficiently, it is desirable to quiesce the system.

Some methods for quiescing a distributed architecture have significantdisadvantages. For example, a distributed architecture may be quiescedby suspending the creation of new messages for a predetermined amount oftime. There are several disadvantages, however, with this method. If allof the messages have not terminated within the predetermined timeperiod, then the state of the distributed system will be destroyedduring the quiescing time period. To avoid this problem, a time periodmay be chosen that, for instance, lasts as long as the time it takes toprocess the longest message (and, perhaps, that message's childrenmessages). Thus, the system assumes that during the time period all ofthe messages will have finished being processed. However, even if thetime period allows all of the messages to terminate, it is likely thatin some cases, all of the messages will have terminated before the endof the predetermined time period, resulting in unnecessary delay. Hence,there is a need for a quiescing system that both ensures that themessages in a distributed architecture have all been processed duringthe quiescing protocol and that terminates without delay when themessages have finished processing.

The systems and methods of the present invention provide a quiescingprotocol. In one embodiment, nodes of a system utilize the protocol tocomplete processing until they reach a consistent state. In oneembodiment, a coordinator initiates the quiescing process and the nodescommunicate with each other to determine whether their messages havebeen processed and communicate with the coordinator to determine whenall of the messages have been processed.

A. Coordinator

The coordinator is responsible for initializing the quiescing protocol,collecting information about the messages the nodes have processed, andterminating the protocol when all of the messages have been processed.

B. Participants

The participants receive messages, process messages, communicate withthe participants to determine whether the messages they sent to theother participants have been processed, and communicate with thecoordinator to inform the coordinator as to the status of the messageprocessing.

C. Communication

In one embodiment, the nodes are interconnected through a bi-directionalcommunication link where messages are received in the order they aresent. In one embodiment, the link comprises a “keep-alive” mechanismthat quickly detects when nodes or other network components fail, andthe nodes are notified when a link goes up or down. In one embodiment,the link comprises a TCP connection. In other embodiments, the linkincludes an SDP connection over Infiniband, a wireless network, a wirednetwork, a serial connection, IP over FibreChannel, proprietarycommunication links, connection based datagrams or streams, and/orconnection based protocols.

D. Quiescing Protocol

In one embodiment, the coordinator sends a message to all participantsindicating that the quiescing protocol is to begin. Each participantprevents new messages from being created. For example, functions thattry to generate new messages are suspended for the duration of thequiescing protocol. The participants then begin tracking the number ofmessages they receive as well as whether messages they have sent outhave been processed.

To track the number of messages they receive, in one embodiment, eachparticipant sets a counter to zero. Thereafter, each participantincrements its own counter every time the participant receives amessage. The participants then process the messages in the order thatthey are received.

To determine whether messages they have sent out have been processed, inone embodiment, the participants begin a probing cycle. Each participantsends out a probe message to every other participant as well as itself.The probe messages are received in each participant's queue. Becausemessages are processed in the order that they are received, once a probemessage is processed, it can be assumed that all messages previouslysent to that same participant and placed in the queue before the probemessage have been processed. When a participant processes a probemessage that it has received, it then sends a response message to thesender. Accordingly, when the probe-sending participant receives aresponse message, the probe-sending participant knows that all messagessent to the responding participant before the probe message was senthave been processed.

After each of the participants has sent out probe messages and receivedthe corresponding response messages, each participant sends a checkpointmessage to the coordinator.

After the coordinator receives checkpoint messages from theparticipants, it sends the participants messages to tell them tocontinue. After receiving the continue message from the coordinator,each participant sends the contents of its counter to the coordinator.After receiving responses from the participants, the coordinator decidesif the subsystem is quiesced or not. If all of the counters are zero,then the subsystem is quiesced and the coordinator sends a done messageto the participants. On the other hand, if any counter is non-zero, thenthe subsystem is not quiesced and the coordinator sends a reset messageto the participants indicating that they need to reset their countersand begin the probing cycle again. By waiting until all of theparticipants have received their response messages and have notprocessed any additional messages, the coordinator knows that all of theoutstanding messages have been processed, and that the system has beenquiesced in a consistent state.

While the embodiments discussed above include the generation of N² probemessages and N² probe responses for a fully connected network. It isrecognized that in other embodiments, other methods may be used toensure that all of the messages sent have been handled between the timethe counter is reset and the time the checkpoint message is sent to thecoordinator.

For example, in one embodiment, the quiescing protocol may include thefollowing:

(1) Reset the counter;

(2) Make sure all messages sent before the beginning of this step havebeen processed by their destination;

(3) Perform a checkpoint;

(4) Perform a second checkpoint, sending along the value of the counter;and

(5) Possibly go back to (1).

Moreover, the determination of whether the messages sent have beenprocessed by their destination may be implemented using probe messagesas discussed above, though it is recognized other approaches may beused, such as, for example, a token approach, a logical-time-basedapproach, a timestamp-based approach, and so forth. In addition, theseother embodiments may be used in connection with a fully connectednetwork as well as with other network topologies as discussed in moredetail below.

The quiescing protocol may be used for a variety of systems that involvemultiple nodes. For example, this protocol may be used in distributedfile systems as described in U.S. patent application Ser. No. 10/007,003entitled “Systems and Methods for Providing a Distributed File SystemUtilizing Metadata to Track Information About Data Stored Throughout theSystem,” filed Nov. 9, 2001 which claims priority to Application No.60/309,803 filed Aug. 3, 2001, U.S. patent application Ser. No.10/281,467 entitled “Systems and Methods for Providing A DistributedFile System Incorporating a Virtual Hot Spare,” filed Oct. 25, 2002, andU.S. patent application Ser. No. 10/714,326 entitled “Systems AndMethods For Restriping Files In A Distributed File System,” filed Nov.14, 2003, which claims priority to Application No. 60/426,464, filedNov. 14, 2002, all of which are hereby incorporated by reference hereinin their entirety.

II. EXEMPLARY QUIESCING SYSTEM

FIG. 1A illustrates one embodiment of an exemplary quiescing system thatincludes a cluster 100 of nodes 110, organized into a fully connectedtopology. A fully connected network is a network where each of the nodesin the network is connected to every other node in the network. Theexemplary cluster 100 comprises six nodes 110. These nodes are,respectively, N₁, N₂, N₃, N₄, N₅, and N₆. The exemplary cluster 100includes a node that has been designated as the coordinator 120. In theillustrated embodiment, the coordinator 120 is designated as N₆. Thecoordinator 120 is responsible for initiating the quiescing protocol.Accordingly, the coordinator 120 communicates with the nodes 110 toinitiate the quiescing protocol and to determine when the quiescingprotocol is complete.

Although in the illustrated embodiment the nodes are arranged in a fullyconnected network topology, in other embodiments of the invention, thenetwork of nodes may be arranged in any other topology, including, butnot limited to, the following topologies: ring, mesh, star, line, tree,bus topologies, and so forth. It will be appreciated by one skilled inthe art that various network topologies may be used to implementdifferent embodiments of the invention. In addition, it is recognizedthat the nodes may be connected directly, indirectly, or a combinationof the two, and that all of the nodes may be connected using the sametype of connection or one or more different types of connections. It isalso recognized that in other embodiments, a different number of nodesmay be included in the cluster, such as, for example, 2, 16, 83, 6883,10,000 and so forth.

The exemplary coordinator 120 includes a coordinator process thatinitiates the quiescing protocol. In one embodiment, the coordinatorprocess lies dormant until its node is designated as the coordinator 120at which time the coordinator process becomes active. In one embodiment,all of the nodes 110 in the cluster 100 include the coordinator processsuch that any one of the nodes in the cluster 100 may be selected as thecoordinator 120. In other embodiments, only a subset of the nodes 110 inthe cluster 100 includes the coordinator process such that only thenodes in the subset may be selected as the coordinator 120.

The coordinator may be selected from the nodes 110 in the cluster 100using a variety of techniques. In one embodiment, the coordinator may bedetermined by a predetermined algorithm or set of conditions. In otherembodiments, the coordinator may be determined dynamically in real timebased on certain parameters or may be determined at random. For example,the coordinator may be determined based on which node first recognizes achange in the cluster, such as when a node has lost its connection withthe other nodes or if a new node has been added to the cluster. Asanother example, the coordinator may be selected based on a round robinprotocol or based on the current utilization of the node's processingpower.

Although in the illustrated embodiment only one node acts as thecoordinator, it is recognized that embodiments may be used wherein twoor more nodes act as the coordinator. In these embodiments, thecoordinator may be implemented as a conceptual entity that operates inthe distributed system on multiple nodes. One skilled in the art willappreciate that there are many ways to configure the system to implementa coordinator process on a cluster of nodes.

FIG. 1B illustrates another embodiment of a fully connected network thatimplements a quiescing protocol. The illustrated cluster 150 is asmaller cluster of nodes 160, than the cluster 100 described above withreference to FIG. 1A. In this embodiment, there are only three nodes160. They are designated, respectively, as N₁, N₂, and N₃. Similar tothe embodiment illustrated in FIG. 1B, the cluster 150 has designatedone of the nodes 160, in this case N₁, as the coordinator 170. Thissimple three-node embodiment will form the basis for the timing chartillustrated in FIG. 2 and described below.

III. EXEMPLARY TIMING CHART

FIG. 2 illustrates a timing chart 200 of one embodiment of messages sentduring a quiescing protocol. The embodiment illustrated in FIG. 2corresponds to the cluster 150 of nodes 160 illustrated in FIG. 1B. Theexemplary timing chart 200 illustrates the flow of messages amongprocesses implementing a quiescing protocol. In the illustratedembodiment, a coordinator process 202 and three participant processes,respectively P₁ 204, P₂ 206, and P₃ 208, exchange messages during thequiescing protocol, where the coordinator process 202 and theparticipant process P₁ 204 are located on N₁, the participant process P₂206 is located on N₂, and the participant process P₃ 208 is located onN₃.

Although in the illustrated embodiment the participant processes resideon different nodes, in other embodiments two or more of the participantprocesses may reside on the same node. For example, the participantprocesses may be separate conceptual entities implemented as differentinstructions and/or memory modules of the same computing device ormemory space.

The exemplary coordinator process 202 and the participant process P₁ 204reside on the same node, N₁. Because they reside on the same node, thesending and receiving of messages between the coordinator process 202and the participant process P₁ 204 may include transactions occurringentirely within the node, such as transactions over an available databus. The participant processes P₁ 204, P₂ 206, and P₃ 208 reside onseparate nodes. When the coordinator process and the participant processreside on different nodes, the communication between them may comprisedata transmissions over a network, using the network protocol TCP/IP,for example. These processes communicate with each other via a networkconnection between the different nodes. One skilled in the art willappreciate that there are many ways to connect nodes and to implementcommunication protocols for the transmission of data between separatenodes or different processes on the same nodes.

The organization of messages sent between the coordinator and theparticipant processes will now be described. In some embodiments of thequiescing protocol the processes may execute certain operations that donot cause messages to be sent to other processes. For the purpose ofthis illustration, those operations are omitted from the exemplarytiming chart. They will be described in detail below with reference toFIGS. 4, 5A, 5B, and 5C.

In the illustrated embodiment, the quiescing protocol is initiated whenthe coordinator process 202 sends a begin message 210 to all of theparticipant processes 204, 206, 208. After receiving the begin message210, each participant process prevents new messages from being createdand sets its respective message counter to zero (not shown). Eachparticipant process then sends a probe message to every participantprocess, including itself. For example, P₃ sends probe messages 212 toP₂ 206 and to P₁ 204, and also to itself 208; P₂ sends probe messages216 to P₁ 204, P₃ 208, and also to itself 206; and P₁ 204 sends probemessages 220 to P₂ 206, P₃ 208, and also to itself 204.

After receiving a probe message, each participant processes the messagesin their respective queues until each participant reaches the probemessage. The participant then processes the probe message and sends aresponse message to the probe-sending participant. For example, afterreceiving the probe message 212 from P₃ 208, P₁ 204, P₂ 206, and P₃ 208process the messages they had received before receiving the probemessages 212, process the probe messages 212, and send response messages214 to P₃ 208. Similarly, after receiving the probe messages 216 from P₂206, each participant process, including P₂ 206, sends a responsemessage 218 to P₂ 206. Finally, after receiving the probe messages 220,each participant process, including P₁ 204, sends a response message 222to P₁ 204. While the response messages occur in the exemplary timingchart soon after the probe messages are received, it is recognized thatin some embodiments, it may take more time, depending on the number ofpending messages, before the response message is sent.

While the timing chart illustrates P₃ first sending its probe messagesand sending them in a sequential order, it is recognized that, in otherembodiments, the participants may send one or more of their probemessages simultaneously, sequentially, non-sequentially, or acombination thereof. For example, P₃ 208 may send its probe messages toP₁ 204, P₂ 206, and itself 208 simultaneously or send them in adifferent order, such as, for example, to P₁ 204, itself 208, and thenP₂ 206 or to P₁ 204 and P₂ 206 simultaneously and then to itself 208.Moreover, while the exemplary participant process P₂ 206 sends its probemessages after participant process P₃ 208 receives response messagesfrom all of the other participants (and P₁ 204 does not send its probemessage until after P₂ 206 receives response messages to its probes),the probe messages may be sent in a different order, such as,simultaneously, sequentially, non-sequentially, or a combinationthereof. Participant processes may send probe messages at any time afterreceiving the begin message 210 from the coordinator process 202. Forexample, P₁ 204 and P₂ 206 may send their probe messages simultaneouslyfollowed by P₃ 208. Similarly, response messages may be sent andreceived simultaneously, sequentially, non-sequentially, or acombination thereof. It will be appreciated by one skilled in the artthat there are many ways to configure participant processes to sendprobe messages and to receive response messages from participantprocesses. Each participant may send its probe messages independent ofthe activities of other participants. Similarly, each participant, afterreceiving a probe message, may send its response message to theprobe-sending participant independent of the probing activities of otherparticipants or the receiving activities of any participant.

After receiving response messages from every participant process,including itself, the participant processes P₁ 204, P₂ 206, and P₃ 208send checkpoint messages 224 to the coordinator process 202. Although inthe illustrated embodiment the participant processes do not send theircheckpoint messages 224 until after every participant process hasreceived its respective response messages, in other embodiments, eachparticipant process may send a checkpoint message to the coordinatorprocess 202 after receiving every response message corresponding to thatparticipant's respective probe message, independent of whether otherparticipants have received response messages, or even sent probemessages. For example, participant process P₃ 208 might send acheckpoint message after receiving response messages 214. Thus, in thisalternative example, P₃ 208 might send its checkpoint message before P₂206 sends its probe message. Alternatively, P₃ 208 might send acheckpoint message after P₂ 206 has sent its probe messages, but beforeP₂ 206 has received response messages, and so forth.

Once the coordinator process 202 receives checkpoint messages 224 fromeach participant process, the coordinator process 202 sends continuemessages 226 to each participant process. After receiving the continuemessages 226 from the coordinator process 202, the participant processessend the contents of their counters 228 to the coordinator process 202.Although in the illustrated embodiment the participant processes sendtheir respective counters 228 at different times, other embodiments maybe used. For example, the participant processes might send theirrespective counters simultaneously with one or more other participantprocesses.

Next, the coordinator process 202 may send either a reset message or adone message to the participant processes. As will be described in moredetail below with reference to FIGS. 4 and 5, the coordinator process202 determines whether to send the reset message or the done messagebased on the contents of the counters 228 received from the participantprocesses. In the illustrated example, the coordinator process 202 sendsthe reset message 230 to each respective participant process, indicatingthat the system has not been completely quiesced. After receiving thereset messages 230, the participant processes begin the probing cycleagain. Thus, the participant processes send another round of probemessages 232 to every participant process. The probe messages 232 may besimilar to or the same as the probe messages 212 sent when the protocolinitiated. Thereafter, the probing cycle may follow the same generalpattern as described above.

Once the coordinator process 202 determines that all of the relevantmessages in the system have been processed, the coordinator process 202sends done messages 236 to the participant processes indicating the endof the quiescing protocol.

Although the timing chart illustrates one embodiment of a quiescingprotocol, it is recognized that there are other embodiments andimplementations of the quiescing protocol. For example, in theillustrated embodiment, participant process P₃ 208 is the firstparticipant process to send a probe message. In other embodiments, theparticipant processes may send probe messages simultaneously.Alternatively, other participant process, such as P₂ 206 or P₁ 204, maybe the first participant process to send a probe message.

IV. QUIESCING SYSTEM NODE

FIG. 3 illustrates the components of one embodiment of a node configuredto implement a quiescing protocol. The exemplary node 300 comprises aprocessor 302 and a quiescing module 304. As illustrated, the processor302 and the quiescing module 304 communicate with one another and maycommunicate with modules outside of the node. In order to implement thequiescing protocol, the quiescing module 304 may include multipleprocesses. In one implementation, the quiescing module 304 may comprisea coordinator process 306 and a participant process 308 which arediscussed below in more detail. In other embodiments, the quiescingmodule 304 may comprise other processes, such as a message-handlingprocess and a message-sending process. These processes comprisedifferent sets of instructions and associated data that executedifferent logical portions of the quiescing protocol.

Although the exemplary node 300 comprises a quiescing module 304 withboth a coordinator process 306 and a participant process 308, in otherembodiments a quiescing module 304 for a particular node may compriseonly a coordinator process 306 or only a participant process 308.Alternatively, a quiescing module 304 may comprise other processes notillustrated here. Moreover, quiescing modules 304 may be configureddifferently for each node, some with coordinator processes 306 andothers with participant processes 308, or with additional processes notillustrated here, or any combination of the same, and so forth.

In one embodiment, the nodes 300 may run on a variety of computersystems such as, for example, a computer, a server, a smart storageunit, and so forth. In one embodiment, the computer may be a generalpurpose computer using one or more microprocessors, such as, forexample, a Pentium processor, a Pentium II processor, a Pentium Proprocessor, a Pentium IV processor, an xx86 processor, an 8051 processor,a MIPS processor, a Power PC processor, a SPARC processor, an Alphaprocessor, and so forth. The computer may run a variety of operatingsystems that perform standard operating system functions such opening,reading, writing, and closing a file. It is recognized that otheroperating systems may be used, such as, for example, Microsoft® Windows®3.X, Microsoft® Windows 98, Microsoft® Windows® 2000, Microsoft®Windows® NT, Microsoft® Windows® CE, Microsoft® Windows® ME, Palm PilotOS, Apple® MacOS®, Disk Operating System (DOS), UNIX, IRIX, Solaris,SunOS, FreeBSD, Linux®, or IBM® OS/2® operating systems.

As described above, the exemplary node 300 includes a quiescing module304. As used herein, the word module refers to logic embodied inhardware or firmware, or to a collection of software instructions,possibly having entry and exit points, written in a programminglanguage, such as, for example, C or C++. A software module may becompiled and linked into an executable program, installed in a dynamiclink library, or may be written in an interpreted programming languagesuch as BASIC, Perl, or Python. It will be appreciated that softwaremodules may be callable from other modules or from themselves, and/ormay be invoked in response to detected events or interrupts. Softwareinstructions may be embedded in firmware, such as an EPROM. It will befurther appreciated that hardware modules may be comprised of connectedlogic units, such as gates and flip-flops, and/or may be comprised ofprogrammable units, such as programmable gate arrays or processors. Themodules described herein are preferably implemented as software modules,but may be represented in hardware or firmware.

V. QUIESCING NODE PROCESSES

In one embodiment, the quiescing module 304 includes a coordinatorprocess 306 and a participant process 308. As described above withreference to FIGS. 1 through 3, these processes may execute on the samenode 300 in the system.

A. Coordinator Process

FIG. 4 illustrates a flowchart of one embodiment of a coordinatorprocess 306 for implementing a quiescing protocol. In state 402, thecoordinator process 306 sends begin messages to the participants. Theparticipants may be processes executing on all of the nodes in a systemor on only a subset of the nodes. In state 404, the coordinator process306 receives checkpoint messages from the participants. As illustratedin state 406, the coordinator process 306 determines whether thecheckpoint messages have been received. If the coordinator process 306is still waiting for checkpoint messages, the coordinator process 306continues to receive checkpoint messages from participants 404, untilthe coordinator process 306 determines that the checkpoint messages havebeen received 406. In state 408, the coordinator process 306 sendsmessages to the participants to continue.

Next, in state 410, the coordinator process 306 receives counters fromthe participants. The counters may include the contents of countingvariables maintained by the participants or a flag variable. Asdiscussed above, when a participant process receives a relevant messageduring the quiescing protocol, the participant process increments itscounter. Similar to states 404 and 406, states 410 and 412 operate aloop until all counters have been received. When the coordinator process306 determines that all counters have been received 412, then thecoordinator process 306 determines whether all of the counters from theparticipant processes are zero 414. If any of the counters from theparticipant processes have a non-zero value, then the coordinatorprocess 306, sends reset messages to the participants 416, and begins towait for checkpoint messages triggered by the probing cycle that resetin each participant process 404. This cycle may repeat as many times asnecessary to allow the existing messages to be processed by the system.Eventually, the probing cycle will yield a collection of counters thatall have the value of zero indicating that all messages have beenprocessed. Once the coordinator process 306 determines that all of thecounters are zero 414, the coordinator process 306 sends done messagesto the participants 418.

In another embodiment, the coordinator process 306 takes the countervalues from the participants, sums the values, and returns the sum tothe participants. If the sum is not zero, then the participants know toreset their counters and restart the probing cycle. If the sum is zero,then the participants know that all of the messages have been processedby the system such that the system is ready to be disabled. In anotherembodiment, the participant processes send a Boolean value, indicatingwhether or not their respective counters are zero, rather than sendingthe actual count of their counter variables. In embodiments where thecounters are implemented as Boolean values, the participants may sendthese values, and the coordinator process 306 may sum the values orcheck each individual Boolean counter to determine if a relevant messagewas counted. It will be appreciated that there are many variations forkeeping track of whether relevant messages were received during thequiescing protocol.

It is recognized that FIG. 4 illustrates one embodiment of a coordinatorprocess 306 and that other implementations of the coordinator process306 may be used.

B. Participant Process

FIGS. 5A, 5B, and 5C illustrate embodiments of combined operations of aparticipant process 308 during the quiescing protocol. FIG. 5Aillustrates one embodiment of a probe-sending process of the participantprocess. FIGS. 5B and 5C illustrate embodiments of other operations thatimplement the quiescing protocol. These operations respond to themessages and activities of participants. One or more of the operationsillustrated in FIGS. 5B and 5C may be implemented as part of anotherprocess, such as, for example, a message-handling process. In someembodiments, the participant process may be broken into severaldifferent processes, including a message-handling process and amessage-sending process. In the illustrated embodiment, these operationsare grouped together into a single participant process that runs atleast in part in parallel, though other embodiments may be used. Theoperations are illustrated separately to simplify the logical flow ofthe component operations.

1. Probe-Sending Process

FIG. 5A illustrates one embodiment of a probe-sending process of aparticipant process. With reference to FIG. 5A, the participant process308 receives messages from the coordinator indicating that the quiescingprotocol is to begin 502. Next, in state 504, the participant process308 suspends the generation of relevant messages for the duration of thequiescing protocol. Messages may be categorized as either relevantmessages or nonrelevant messages. Relevant messages are those messagesthat change the state of the system. Relevant messages are suspendedbecause messages that change the state of the system are the messagesthat need to be stabilized during the quiescing protocol. Once therelevant messages have been completed, then the system has reached aconsistent state. In one embodiment, the messages initiated by thequiescing protocol are non-relevant messages because they do not changethe state of the system.

The following illustrates an example probing cycle 524. In state 506,the participant process 308 sets its counter to zero. As described inmore detail below with reference to FIG. 5C, when a relevant message isreceived by the participant process 308, the participant process 308increments its counter by one, though in other embodiments, a flag, suchas a binary flag, may be used. Then, in state 508, the participantprocess 308 sends probe messages to the participants, including itself.As described in detail, with reference to FIG. 2, the probe messages maybe sent to the participants in a variety of combinations. After sendingprobe messages to the participants, each participant process 308 waitsto receive all of the response messages from the participants. Thiswaiting period is illustrated in states 510 and 512. In state 510, theparticipant process 308 receives responses from the participants. Then,in state 512, the participant process 308 determines whether theresponse messages have been received from the participants. If they havenot all been received, the participant process 308 returns to state 510,waiting for additional response messages from the remainingnon-responding participants.

Once the participant process 308 has determined that the responsemessages have been received, the participant process 308 sends acheckpoint message to the coordinator 514. Then, in state 516, theparticipant process 308 receives a continue message from thecoordinator. In some embodiments there may be some delay between thesending of the checkpoint message and the receiving of the continuemessage because the coordinator does not send the continue message untilit has received checkpoint message from all of the participants. Instate 518, the participant process 308 sends the contents of its counterto the coordinator. Then, in state 520, the participant process 308waits to receive either the done message or the reset message from thecoordinator. If the coordinator has determined that the system isquiesced, then the coordinator sends the done message, indicating to theparticipant processes that the quiescing protocol is finished. If,however, the coordinator determines that the system is not quiesced 522,then the coordinator sends a reset message to the participant processes,indicating that the probing cycle should be repeated. If the participantprocess 308 receives the reset message from the coordinator, then theparticipant process returns to state 506, setting its counter to zero,and then proceeds to send probe messages to the participants 508, and soforth. The operations from state 506 through 522 are referred to as theelements of the probing cycle 524. Once the participant process 308receives the done message from the coordinator, the probing cycle isexited.

As set forth above, in other embodiments, the participant process 308may receive from the coordinator a sum of the counters of all of theparticipants. If the sum is not zero, then the participant process 308returns to state 506, setting its counter to zero, and then proceeds tosend probe messages to the participants 508, and so forth. If the sum iszero, then the participant process exits the probing cycle and proceedsto the end state.

2. Probe-Response Process

FIG. 5B illustrates one embodiment of a probe-response process of aparticipant process 308. FIG. 5B illustrates a probe-response processthat the participant process executes in addition to the probe-sendingprocess described above with reference to FIG. 5A. This flowchartillustrates the operations taken by each participant process when itreceives a probe message from a participant process, including itself.In one embodiment, the probe-response process may run in parallel withthe probe-sending process.

In state 532, the participant process 308 receives a probe message froma participant, including itself. In state 534, the participant process308 determines whether all relevant messages from the probe-sendingparticipant have been handled. If all of the relevant messages from theprobe-sending participant have been handled, as illustrated in decisionstate 536, then the participant process 308 sends a response message tothe probe-sending participant, as illustrated in state 538. If, on theother hand, the participant process 308 determines that all of therelevant messages from the probe-sending participant have not beenhandled, then the participant process 308 proceeds back to state 534where it determines again whether all of the relevant messages from theprobe-sending participant have been handled. This loop continues untilthe participant process 308 determines that all of the relevant messagesfrom the probe-sending participant have been handled. The effect of theprobe-response process described above is to ensure that responses toprobe messages are not sent until the participant process 308 hashandled all of the relevant messages from the probe-sending participant.

In one embodiment, messages are received into a queue and processed inthe order that they are received. Thus, if a participant process 308processes messages from a queue and receives a probe message, it can beassumed that all messages received from the probe-sending participantbefore the probe message was sent have been handled since all messagesin the queue received before the probe message have been handled.

As set forth above, while the probing cycle is described above, it isrecognized that in other embodiments, other methods may be used toensure that all of the messages sent have been handled between the timethe counter is reset and the time the checkpoint message is sent to thecoordinator.

3. Counter Process

FIG. 5C illustrates one embodiment of a counter process of theparticipant process. FIG. 5C illustrates how each participant process308 handles the receipt of relevant messages during the quiescingprotocol. In state 562, the participant process 308 receives a messagefrom a participant, which may be itself. After receiving a message froma participant, the participant process 308 determines whether themessage is a relevant message 564. If it is a relevant message, then theparticipant process 308 increments its counter 566 or sets its flag.After incrementing its counter, the participant process 308 processesthe message 568. If, however, the message is determined not to be arelevant message, then the participant process 308 processes the messagewithout incrementing its counter.

In other embodiments, rather than process the message, the message isadded to the queue to be processed. In other embodiments, the counterprocess may count the relevant messages sent, rather than the relevantmessages received. In this embodiment, the participants would countrelevant messages sent after their counters were reset at the initiationof the probing cycle.

In some embodiments, the counter process may also process messagesreceived from non-participant nodes and/or other components of thesystem. Moreover, the counter process may operate during differentperiods of time, according to different embodiments. For example in someembodiments, the counting process may run in conjunction with theprobing cycle, as described above with reference to FIG. 5A. In otherembodiments, the counting process may operate in parallel during theexecution of the entire quiescing protocol, or even before and/or afterthe quiescing protocol.

It will be appreciated by one skilled in the art that there are manyways to implement the processes described in FIGS. 5A, 5B, and 5C. Forexample, the processes may be implemented as subroutines or as separateprogramming threads.

VI. EXAMPLE APPLICATION

An example application of the quiescing protocol will now be describedin the context of a distributed file system. The exemplary distributedfile system includes nodes and resources. Since all nodes in a clustershare the same resources, they need to make sure that their use of aresource does not conflict with any others cluster-wide. Accordingly,each resource is assigned an ID as well as a resource coordinator thatcoordinates access to that particular resource. These assignments arestored as a mapping that maps the resource's ID to a resourcecoordinator.

To control access to the resources, the resource coordinators employ a“locking” protocol. To use a resource, a client thread first acquires alock on that resource. When a client thread needs a lock, it will usethe mapping to find the resource coordinator for the requested resourceand send a lock request to that resource coordinator.

Problems may occur, however, when a change needs to be made to themapping. Changes to the mapping may occur, for example, when nodes areadded to or removed from the cluster. The mapping cannot just change inan ad hoc fashion since at any point in time there are lock requeststhat are waiting to be processed and lock requests that are beingcreated that use the old mapping. Accordingly, in one embodiment, thecluster may be quiesced such that all new lock requests are suspended,all pending lock requests are processed, the old mapping is replacedwith the new mapping, and the new lock requests resume using the newmapping. In order to ensure that the new lock requests are suspended andall pending lock requests are processed, the cluster may implement thequiescing protocol discussed above.

A. Removal of a Node

FIGS. 6A, 6B, and 6C illustrate embodiment of the removal of a node froma cluster, which prompts a change to reassign the resource for which theremoved node had been the coordinator. FIG. 7 illustrates a flowchartdescribing the steps taken to execute a group change, including thequiescing of the system.

FIG. 6A illustrates one embodiment of a smart storage unit cluster 600.The cluster 600 is arranged in a fully connected topology. The cluster600 comprises six nodes: N₁ 602, N₂ 604, N₃ 606, N₄ 608, N₅ 610, and N₆612. Each node is also associated with a corresponding resource,respectively: R₁ 622, R₂ 624, R₃ 626, R₄ 628, R₅ 630, and R₆ 632. In theillustrated embodiment, each node is the coordinator for its associatedresource. Although in the illustrated embodiment each node is associatedwith exactly one resource, in other embodiments one node may beassociated with multiple resources or with no resources at all. Thus, insome embodiments, some nodes may be the coordinator for multipleresources and/or some nodes may not be coordinators for any resource. Inaddition, some resources may be associated with one or more nodes. Amore detailed description of various embodiments of smart storage unitare disclosed in the patent applications incorporated by referenceabove.

FIG. 6B illustrates one embodiment of the status of the interconnectionsof a cluster 600 during a group change. In the illustrated embodiment,smart storage unit N₃ 606 is removed from or loses communication withthe smart storage unit cluster 600. The dashed lines indicate theremoval of the smart storage unit N₃ as well as the break incommunication with the remaining smart storage units. Once smart storageunit N₃ has been removed from the cluster 600, the remaining nodesreassign the responsibility for resource R₃ 626, which was previouslymanaged by N₃ 606. This reassignment of responsibilities is called agroup change.

FIG. 6C illustrates one embodiment of the makeup of cluster 600,following the group change. The cluster now comprises five smart storageunits, respectively N₁ 602, N₂ 604, N₄ 608, N₅ 610, and N₆ 612. Four ofthe nodes in the cluster 600 retain their original assignments ascoordinators for the respective resources. Smart storage unit N₂ 604,however, has been reassigned as the coordinator for R₃. Thus, smartstorage unit N₂ 604 is now the coordinator for both R₂ and R₃. In orderto reassign resources from FIG. 6A to 6C, it is preferable that thesystem is quiesced before such changes are made.

B. Mapping Change

FIG. 7 illustrates one embodiment of a process 700 for implementing achange to the resource assignments. In state 702, all available nodesagree that a group change is taking place. Then, in state 704, all nodesstop using the old mapping function and new lock requests are suspended.Then, in state 706, the cluster is quiesced so that any outstandingmessages using the old mapping structure are delivered and handled.States 704 and 706, collectively, represent the quiescing protocol 708.Thus, the flowcharts described above with reference to FIGS. 4, 5A, 5B,and 5C may be used to execute states 704 and 706 in an exemplary groupchange application. Once the quiescing protocol has been executed, instate 710, the cluster redistributes the lock manager state according tothe new mapping function. Finally, in state 712, all lock requests areresumed using the new mapping function.

C. Exemplary Code

The following is exemplary code for implementing a change to theresource assignments.

static int rtxn_sync_locks_prepare(const struct gmp_info *old_gi, conststruct gmp_info *new_gi, enum rtxn_sync_locks_state *state) { int cont,error; u_int64_t lk_recv_count; /* Suspend initiator side. New messageswill be generated by  * lk_lock, lk_unlock, etc. However, messages inresponse to  * other messages will still be generated. */lk_group_change_suspend_api( ); /* Drain locks. */ cont = 1; do { /*Remember the old receive count. */ lk_recv_count = lk_get_recv_count( );/* Do probe. */ error = rtxn_probe(&new_gi->up_nodes); if (error) {INOTICE(“error %d from %s”, error, function_name(rtxn_probe));rtxn_abort( ); return 0; } /* Do checkpoint. */ if (!rtxn_checkpoint(0))return 0; /* Do another checkpoint, this time sending * whether or notwe received any messages. * The coordinator will sum all of the'cont's * that it receives from participants. */ cont = lk_recv_count !=lk_get_recv_count( ); if (!rtxn_checkpoint(&cont)) return 0; } while(cont); return 1; } u_int64_t lk_get_recv_count(void) {mtx_lock(&g_lk.lock); u_int64_t ret = g_lk.recv_count;mtx_unlock(&g_lk.lock); return ret; } void handle_lk_lock_request(structdev_local_ctx *dl_ctx, enum lk_domain_type domain, const u_int8_tresource[ ], u_int type, int waiter_token) { struct lk_domain *ld =&lk_domains[domain]; mtx_lock(&g_lk.lock); g_lk.recv_count++;lk_recv_lock_request(ld, dl_ctx->devid, resource, type, waiter_token);mtx_unlock(&g_lk.lock); } void handle_lk_lock_response(structdev_local_ctx *dl_ctx, enum lk_domain_type domain, const u_int8_tresource[ ], const struct int_set *tokens) { struct lk_domain *ld =&lk_domains[domain]; mtx_lock(&g_lk.lock); g_lk.recv_count++;lk_recv_lock_response(ld, dl_ctx->devid, resource, tokens);mtx_unlock(&g_lk.lock); }

VII. CONCLUSION

While certain embodiments of the invention have been described, theseembodiments have been presented by way of example only, and are notintended to limit the scope of the present invention. Accordingly, thebreadth and scope of the present invention should be defined inaccordance with the following claims and their equivalents.

1. A distributed system configured to quiesce a set of messages, thedistributed system comprising: a plurality of nodes, each nodecomprising one or more processors; a first subset of two or more of theplurality of nodes, each node of the first subset further comprising aparticipant process; a second subset of one or more of the plurality ofnodes, each node of the second subset further comprising a coordinatorprocess; and a set of messages sent and received by the plurality ofnodes, the set of messages comprising: a relevant message which changesa state of the distributed system; a probe message which requests aprobe-response message; the probe-response message which indicates thatthe sender has processed all received relevant messages from therecipient; a checkpoint message which indicates that the sender hasreceived a probe-response message from each of the plurality of nodes; acontinue message requesting a continue-response message; and thecontinue-response message which indicates whether the sender hasreceived a relevant message from one or more of the plurality of nodes;wherein each participant process is configured to, when executed:suspend generation of relevant messages; maintain received-messageinformation which indicates whether a relevant message has been receivedfrom the plurality of nodes; send probe messages to each of theplurality of nodes; receive probe-response messages from each of theplurality of nodes; receive probe messages from each of the plurality ofnodes; and for each probe message received, send the probe-responsemessage to the node which sent the probe message; and wherein eachcoordinator process is configured to, when executed: receive checkpointmessages from each of the plurality of nodes; send continue messages toeach of the plurality of nodes; receive continue-response messages fromeach of the plurality of nodes; and based on the receivedcontinue-response messages, determine whether the distributed system hasbeen quiesced.
 2. The distributed system of claim 1, wherein the firstsubset and the second subset are the same.
 3. The distributed system ofclaim 2, wherein both the first subset and the second subset compriseall of the plurality of nodes.
 4. The distributed system of claim 2,wherein the participant process and the coordinator process areimplemented as a single module.
 5. The distributed system of claim 1,wherein the first subset and the second subset are different.
 6. Thedistributed system of claim 1, wherein the first subset comprises all ofthe plurality of nodes.
 7. A distributed system configured to quiesce aset of messages, the distributed system comprising: a plurality ofnodes, each node comprising a processor; and one or more executablecoordinator processes, each coordinator process configured to, whenexecuted: receive one or more first messages from one or more of theplurality of nodes, each first message indicating that the node, whichhas sent that first message, has sent a second message to each of theplurality of nodes and has received a third message from each of theplurality of nodes, wherein the second message is a message requestingthe third message, and wherein the third message indicates that allmessages that change a state of the distributed system previouslyreceived by the node, which has received that second message, from thenode, which has sent that second message, have been processed; inresponse to receiving the one or more first messages, send fourthmessages to the plurality of nodes, wherein each fourth message is amessage requesting a fifth message; receive one or more fifth messagesfrom one or more of the plurality of nodes in response to the fourthmessages, each fifth message indicating whether the node, which has sentthat fifth message, has received a message that changes a state of thedistributed system; and based on one or more received fifth messages,determine whether the distributed system has been quiesced.
 8. Thedistributed system of claim 7, wherein each of the plurality of nodesfurther comprises one of the one or more executable coordinatorprocesses.
 9. The distributed system of claim 7, wherein the pluralityof nodes includes at least one coordinator node, the coordinator nodefurther comprising one of the one or more executable coordinatorprocesses.
 10. The distributed system of claim 9, wherein sending thefourth messages includes sending a remote message to at least one of theplurality of nodes, and receiving the one or more first messages andfifth messages includes receiving a remote message from at least one ofthe plurality of nodes.
 11. The distributed system of claim 10, whereinsending the fourth messages further includes sending a local message tothe coordinator node, and receiving the one or more first messages andfifth messages further includes receiving a local message from thecoordinator node.
 12. The distributed system of claim 7, wherein thefifth message comprises information about the number of messages thatchange a state of the distributed system which have been received by thenode which has sent the fifth message.
 13. The distributed system ofclaim 7, wherein the fifth message comprises a Boolean value.
 14. Thedistributed system of claim 7, wherein each coordinator process isconfigured to determine that the distributed system has been quiescedwhen the one or more received fifth messages indicate that the pluralityof nodes have not received a message that changes a state of thedistributed system.
 15. A distributed system configured to quiesce a setof messages, the distributed system comprising: a plurality of nodes,each node comprising a processor and at least one executable softwaremodule; wherein the at least one executable software module of each ofthe plurality of nodes is configured to, when executed: suspendgeneration of new messages that change a state of the distributedsystem; maintain received-message information which indicates whether amessage that changes a state of the distributed system has been receivedfrom the plurality of nodes; send first messages to the plurality ofnodes, each first message requesting a response; receive one or moresecond messages from one or more of the plurality of nodes, each secondmessage indicating that all messages which change a state of thedistributed system sent by the node, which received the second message,to the node, which sent the second message, have been processed; receiveone or more third messages from one or more of the plurality of nodes,each third message requesting a response; and for each third messagereceived, send a fourth message to the node, which sent the thirdmessage, each fourth message indicating that all messages which change astate of the distributed system sent by the node, which sent the thirdmessage, to the node, which received the third message, have beenprocessed.
 16. The distributed system of claim 15, wherein sending thefirst message and fourth messages includes sending a remote message toat least one of the plurality of nodes, and receiving the one or moresecond messages and third messages includes receiving a remote messagefrom at least one of the plurality of nodes.
 17. The distributed systemof claim 15, wherein sending the first messages and fourth messageincludes sending a local message to the node which is sending the firstmessages and fourth message, and receiving the one or more secondmessages and third messages includes receiving a local message from thenode which is receiving the one or more second messages and thirdmessages.
 18. The distributed system of claim 15, wherein thereceived-message information comprises the number of messages thatchange a state of the distributed system which have been received fromthe plurality of nodes.
 19. The distributed system of claim 15, whereinthe received-message information comprises a Boolean value.
 20. Adistributed system comprising: a plurality of three or more nodes, eachnode comprising a processor and configured to act as a coordinator nodeand a participant node in a quiescing protocol; wherein, duringoperation of the quiescing protocol, the plurality of three or morenodes comprises at least one coordinator node and a plurality of two ormore participant nodes; wherein the at least one coordinator node isconfigured to send and receive messages to and from each of theplurality of participant nodes to collect information about messageswhich the plurality of participant nodes have received; and wherein eachparticipant node is configured to: send and receive messages to and fromthe plurality of participant nodes to determine whether messages whichthe participant sent to the plurality of participant nodes and whichchange a state of the distributed system have been processed; and send amessage to the at least one coordinator node to communicate whether amessage that changes a state of the distributed system has been receivedfrom one or more of the plurality of participant nodes during operationof the quiescing protocol.
 21. The distributed system of claim 20,wherein the coordinator node is one of the plurality of two or moreparticipant nodes.
 22. The distributed system of claim 20, wherein thecoordinator node is not one of the plurality of two or more participantnodes.